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ABSTRACT 

Air  Force  (AF)  Major  Commands  (MAJCOMs)  often  have  difficulty  in  understanding  how  much  improved 
war-fighting  capability  can  be  expected  from  investments  in  Manpower,  Personnel,  and  Training  (MPT)  or 
Human  Performance  Research  and  Development  (R&D).  This  is  because  MPT  R&D  findings  usually  have  not 
related  directly  to  war-fighting  Measures  of  Effectiveness  (MOEs).  To  convince  MAJCOMs  of  the  utility  of  their 
research,  MPT  R&D  personnel  must  demonstrate  how  combat  capability  will  be  increased  by  these  investments. 

To  clarify  these  relationships,  CSERIAC  was  tasked  to  identify  MOEs  and  Measures  of  Performance  (MOPs) 
which  could  serve  as  criteria  for  human  system  R&D,  and  to  hierarchically  organize  these  MOEs  for  linkage  to  a 
set  of  improved  training  effectiveness  measures.  Overall,  the  goal  is  to  enable  commanders  to  predict  how,  for 
example,  specified  screening  and  training  technology  can  contribute  to  imit  effectiveness.  In  Phase  I  of  the 
project,  CSERIAC  searched  the  literature,  brainstormed  with  readiness  research  experts,  and  examined 
human-system  interface  criteria  forjudging  mission  effectiveness  for  aeronautical  systems.  We  foimd  that  the 
literature  contains  no  comprehensive  listing  of  human-related  aircraft  MOEs;  therefore  we  compiled  a  listing 
from  the  literature.  In  Phase  n  using  the  MOE  hierarchy,  CSERIAC  will  interview  AF  decision-makers  with 
wartime  missions  and  responsibilities  to  capture  their  criteria  forjudging  unit  readiness  and  effectiveness  for 
comparison  with  the  compiled  MOE  hierarchy.  Measures  prioritized  as  most  important  will  then  be  linked  to 
MPT  effectiveness  metrics.  With  these  linkages,  the  value  of  advanced  training  and  screening  technology  can  be 
more  clearly  demonstrated.  This  paper  will  provide  Phase  I  results  and  highlight  plans  for  the  remaining  project. 

INTRODUCTION 

Defining  the  Problem 

Declining  R&D  Budget  Demands  Research  Targeted  to  Combat  Capability.  MPT  R&D  traditionally 
validates  its  products  and  procedures  using  outcome  criteria  such  as  course  grades,  "time-to-master," 
cost-benefits  analyses,  or  more  recently,  job  performance  indices.  With  major  R&D  program  budgets  declining,  ^ 
MPT  researchers  must  demonstrate  their  contribution  to  overall  mission  effectiveness  to  gain  support  from 
MAJCOM  officials,  and  higher  priority  in  the  budgeting  cycle.  This  project  is  based  on  the  assumption  that 
research  funding  prioritizations  may  be  greatly  influenced  by  demonstrating  war-fighting  capabilities  and 
readiness  impacts.  Therefore,  incorporating  relevant  readiness  and  war-fighting  criteria  into  research  program 
descriptions  and  impact  statements  is  critical  to  future  research. 

MPT  Scientists  Not  Using  Combat  Capability  Criterion  Link.  Unfortunately,  MPT  scientists  have  not 
empirically  shown  this  link,  and  top  decision-makers  may  use  this  perceived  lack  of  evidence  as  an  indicator  of  ■ 
the  relative  merit  of  MPT  R&D  during  today's  "downsizing"  budget  climate.  One  reason  why  research  scientists 
do  not  consistently  use  empirical  links  to  readiness  and  war-fighting  capability  is  that  they  do  not  know  what 
these  higher  level  war-fighting  criteria  are,  or  where  and  how  to  obtain  these  criterion  measures.  They  have  been 
accustomed  to  using  course  and  job  criteria.  Especially  in  these  times,  MPT  effectiveness  measures  must  be 
linked  with  war-fighting  capability  measures.  But  few  MPT  scientists  are  aware  of  what  these  top 
decision-makers  and  potential  users  of  their  technology  will  accept  as  indicators  of  merit,  nor  are  they  are  fully 
aware  of  which  decision-maker  they  should  target  for  demonstrations  of  MPT  R&D  technology  with  what  kind 
of  wartime  links.  To  extend  the  connection  from  these  typical  MPT  measures  to  criteria  that  MAJCOM 
decision-makers  value,  scientists  need  a  taxonomy  to  assist  them  in  identifying  relevant  and  organizationally 
valued  criteria'.  The  basic  problem  is  that  MPT  scientists  do  not  know  the  identity  and  interests  of 
decision-makers  who  might  be  educated  on  the  true  return  on  human  systems  R&D  investments  (Gould,  1995). 

Defining  Key  Terms 

Definition  of  MOEs  &  MOPs.  One  way  to  target  human  performance  improvement  research  is  to  focus  on  the 
Measures  of  Effectiveness  (MOEs)  and  Measures  of  Performance  (MOPs)  used  to  evaluate  the  performance  of 
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weapon  systems,  organizational  tmits,  teams  or  crews,  and  individual  airmen.  MOEs  are  a  measure  of  how  well 
an  operational  task  or  task  element  is  accomplished  through  using  a  system  (ordinarily  a  single  operator  using 
one  aircraft  or  weapons  system  to  perform  one  task)  (AFOTEC/XR.C,  1995;  Lane,  1986;  USAF/TEP,  1994). 
MOPs  are  qualitative  or  quantitative  measures  of  system  capabilities  or  characteristics  (USAF/TEP,  1994).  They 
indicate  the  degree  to  which  that  capability  or  characteristic  performs  or  meets  the  requirement  under  specific 
conditions  (USAF/TEP,  1994).  MOPs  are  components,  or  subsets,  of  MOEs;  i.e.,  the  "degree-to-which"  a  system 
performs  is  one  of  a  number  of  possible  measures  of  "how  well"  a  system's  task  is  accomplished.  Therefore, 
MOPs  can  be  accumulated  to  assess  an  MOE  that  is  not  directly  measurable  (AFOTEC/OQR.C,  1995).  In  these 
definitions,  however,  there  appears  to  be  much  overlap  and  interchangeability  between  MOEs  and  MOPs  (Lane, 
1986),  with  some  MOPs  being  labeled  as  MOEs;  therefore,  we  refer  to  these  terms  with  a  slash  between  them 
(i.e.,  MOE/MOP).  Additionally,  since  the  AF  Operational  Test  and  Evaluation  Center  (AFOTEC)  focuses  on 
total  system  performance,  and  since  the  "total  system"  as  defined  by  Department  of  Defense  (DoD)  Directive 
5000.1  (DoD,  1996,  p.5)  includes  "the  people  who  operate  and  maintain  the  system"  and  their  "training  and 
training  devices,"  the  broader  definition  of  "total  system"  MOEs/MOPs  would  include  the  effectiveness  of  units, 
team  or  crew,  and  individual  performance  associated  with  the  system. 

Other  Essential  Definitions.  Other  important  definitions  include  the  following;  Mission  Effectiveness  is  defined 
as  the  probability  that  a  system  (also  read  this  to  include  the  unit,  team  or  crew,  or  individual)  is  available  to 
initiate  its  mission  and  will  complete  its  mission  when  initiated  (USAF/LGMM,  1994).  War-fighting  (Combat) 
Capability  is  determined  by  the  number  of  consecutive  events  (e.g.,  sorties,  miles,  orbits,  hours)  that  a  weapon 
system  can  successfully  execute  its  mission  under  specified  conditions  (USAFILGMM,  1994).  Readiness  has 
been  defined  as  the  ability  of  forces,  imits,  weapon  systems,  or  other  equipment  to  deliver  the  output  for  which 
they  were  designed,  including  the  ability  to  deploy  and  employ  without  unacceptable  delay  (USAF/XOOO, 
1993). 

Project  Objective  and  Tasks 

Based  on  the  need  to  identify  potential  sponsors  of  MPT  R&D  and  criterion  measures  that  these  decision-makers 
will  accept  as  essential,  the  Armstrong  Laboratory  Technical  Training  Research  Division  (AL/HRT)  tasked 
CSERIAC  to  accomplish  the  following  objective:  Develop  a  hierarchical  taxonomy  of  war-fighting  capability 
metrics  linking  the  priorities  and  preferences  of  AF  operational  command  decision-makers  for  specific  MOEs  or 
MOPs.  The  overall  project  is  divided  into  two  phases,  with  a  CSERIAC  Review  and  Analysis  documenting  the 
results  of  the  first  phase.  The  specific  tasks  and  approach  are  listed  below. 

Phase  I  tasks  include  the  following:  (a)  survey  the  literature  and  network  with  experts  on  aircraft-related 
MOEs/MOPs;  (b)  develop  a  candidate  listing  of  AF  operational  flying  command  officials  responsible  for 
wartime  missions  and  identify  MOEs/MOPs;  (c)  based  on  the  literature  search  and  networking  findings,  develop 
a  consolidated  taxonomy  of  MOEs/MOPs  for  aircraft  systems,  aircrew,  and  maintenance,  and  display  these 
MOEs/MOPs  together  with  their  meaning,  computational  formula,  and  source;  and  (d)  document  the  findings  in 
a  CSERIAC  Review  and  Analysis  report. 

Phase  n  tasks  include  the  following:  (a)  construct  a  structured  interview  based  on  the  findings  in  Phase  I 
designed  to  capture  the  preferred  MOEs/MOPs  and  their  priority  from  MAJCOM  decision-makers;  (b)  develop  a 
taxonomy  of  these  MOEs/MOPs  showing  hierarchical  relationships  of  the  criteria  of  war-fighting  capability 
through  decomposing  those  criteria  associated  with  human  systems  implications;  and  (c)  liiJc  decision-makers 
and  their  preferences. 

Overview 

After  a  review  of  the  approach  taken  in  this  study,  this  paper  reviews  Phase  I  findings.  Leading  sources  of 
MOEs/MOPs  were  found  to  be  military  standards  and  AF  instructions,  mission  area  assessments  and  plans, 
acquisition  documents,  test  and  evaluation  (T&E)  reports,  the  readiness  reporting  systems,  operational  readiness 
inspections,  command  status  briefings,  exercises,  wargames  and  simulations,  competitions,  and  related  surrogate 
measures.  Based  on  these  findings,  recommendations  are  made  for  Phase  II  of  this  study. 

APPROACH 

CSERIAC  conducted  extensive  literature  searches  of  government  and  commercial  databases,  such  as  the  Defense 
Technical  Information  Center  (DTIC),  to  collect  and  organize  backgroimd  information  on  MOEs  and  MOPs  for 
aeronautical  systems,  training,  and  MAJCOMs.  The  main  source  of  MOEs  in  the  scientific  literature  was  T&E 
reports;  however,  most  of  these  reports  were  either  "classified"  or  "limited  distribution"  documents,  and  were  not 
used  for  the  present  search.  Because  there  was  no  comprehensive  listing  of  aircraft-related  MOEs/MOPs, 
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CSERIAC  began  constructing  one.  Since  the  literature  failed  to  contain  summary  articles  on  MOEs/MOPs,  it  was 
necessary  to  begin  extensive  "networking"  with  MAJCOM  officials  to  identify  the  MOEs/MOPs  which  were  not 
listed  in  DTIC.  CSERIAC  requested  MAJCOM  officials  to  provide  information  on  these  "undocumented" 
measures  which  are  being  used  to  determine  the  health  of  their  aircraft  systems  and  evaluate  unit  or  team 
performance  and  force  readiness.  The  resulting  MOE/MOP  information  was  organized  into  a  MOE  listing, 
organized  by  fimctional  and  mission  area. 

FINDINGS 


MOE/MOPs 

Numerous  and  Many  Variants.  While  there  is  a  limited  set  of  AF  missions,  mission  tasks  and  subtasks,  AF 
MOEs  and  MOPs  are  as  numerous  as  there  are  organizations  attempting  to  fulfill  those  missions. 

Not  Well  Documented  in  Literature.  A  major  finding  of  this  literature  search  is  that  because  of  the  tremendous 
variety  and  number  of  MOEs/MOPs,  a  comprehensive  listing  does  not  exist  in  the  literature,  and  is  unlikely  to  be 
constructed  and  maintained.  Further,  MOEs/MOPs  are  not  well  documented  in  the  literature,  and  must  often  be 
inferred  or  derived. 

Sources  of  MOEs/MOPs.  The  following  sources  yielded  significant  MOEs/MOPs: 

MIL-STD  1776A.  The  most  extensive  source  of  aircraft-related  MOEs/MOPs  was  Military  Standard  (MIL-STD) 
1776A  Aircrew  Station  and  Passenger  Accommodations  (DoD,  1994).  It  contains  a  table  of  aeronautical 
MOEs/MOPs  together  with  air  tasks  and  subtasks,  although  it  is  not  so  comprehensive  as  to  include  all  missions, 
tasks,  subtasks,  and  formulas  to  compute  the  MOEs/MOPs.  We  identified  no  other  current  MIL-STD  that 
contained  MOEs/MOPs  to  this  extent. 

AFI 10-602.  The  most  comprehensive  listing  of  aircraft  maintenance-related  MOEs/MOPs  was  found  in  Air 
Force  Instruction  (AFI)  10-602,  Determining  Logistics  Support  and  Readiness  Requirements,  Attachment  3 
(USAF/LGMM,  1994).  This  listing  provides  formulas  to  compute  MOEs/MOPs  like  "maintenance  hours  per 
flying  hours,"  "mean  time  between  failure,"  and  "mean  time  to  repair,"  which  are  the  measures  used  to  ju^e 
success  at  the  AF  wing  organizational  maintenance  units.  These  measures  are  also  used  for  evaluation  purposes 
up  the  chain  of  command  and  for  the  evaluation  of  proposed  acquisitions  and  modifications. 

MAAs  &  MAPs.  A  third  source  of  high-level  MOEs/MOPs  is  the  Defense  Modernization  Planning  process 
(USAF/OCOXP,  1996).  In  this  structured  process  Mission  Area  Assessments  (MAAs)  are  being  developed  for 
each  mission  area.  The  products  of  these  assessments  are  documented  in  Mission  Area  Plans  (MAPs)  which 
include  both  MOEs/MOPs  and  deficiencies  (Air  Mobility  Command  [AMC]/Plans  [XP],  1995).  Also  by 
implication,  one  can  develop  additional  MOEs/MOPs  from  these  deficiencies. 

Acquisition  Documents.  Related  to  the  MAAs  and  MAPs  are  the  Mission  Need  Statements  (MNSs)  and 
Operational  Requirements  Documents  (ORDs)  which  contain  MOEs/MOPs  for  new  or  modified  acquisitions.  In 
the  ORD  a  table  called  the  Requirements  Correlation  Matrix  (RCM)  contains  detailed  MOEs/MOPs  that  are 
requested  by  the  using  MAJCOM  for  the  new  or  modified  system  to  achieve. 

Test  and  Evaluation.  Later  in  acquisition,  the  Test  and  Evaluation  Master  Plan  (TEMP)  is  developed,  detailing 
how  the  system  will  be  tested.  Either  the  AF  Flight  Test  Center  at  Edwards  AFB  or  AFOTEC  will  develop  an 
extensive  hierarchy  of  Critical  Operational  Issues  (COIs),  MOEs,  and  MOPs  to  assess  operational  effectiveness 
and  suitability  of  the  system  (AFOTEC/XRC,  1995).  T&E  reports  were  helpful  in  identifying  some 
MOEs/MOPs;  however,  classified  reports  were  not  examined  to  date.  Remaining  limited  or  classified  T&E 
reports  could  be  examined  more  closely  (time  permitting)  for  hierarchical  relationships  and  MOEs/MOPs 
associated  with  each  weapon  system.  Researchers  may  want  to  examine  TEMPs  which  are  negotiated  with 
MAJCOM  input,  and 

later  in  acquisition,  T&E  reports,  to  determine  what  the  using  command  will  accept  as  proof  that  a  system  is 
meeting  the  requirements  documented  in  the  RCM. 

Readiness  Reporting  Systems.  AF  Policy  Directive  (AFPD)  10-2,  Readiness  (USAF/XOOO,  1993),  requires 
that  commanders  assess  the  readiness  of  their  resources.  The  prime  readiness  reporting  system  is  the  Status  of 
Resources  and  Training  System  (SORTS).  SORTS  is  the  mechanism  for  assessing  the  readiness  or  "C-Rating"  of 
each  operational  unit  and  reporting  it  up  the  chain  of  command.  SORTS  concentrates  on  four  areas  of 
determination:  (1)  personnel,  (2)  equipment  on  hand,  (3)  training,  and  (4)  equipment  condition.  The  AF  version 
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of  SORTS  is  called  the  AF  Integrated  Readiness  Measurement  System  (AFIRMS)  (USAF/XOOOR,  1995; 
SofTech,  1985).  Other  forms  of  readiness  data  and  reporting  systems  have  been  developed  or  tested  to 
supplement  SORTS  because  it  was  considered  by  the  Government  Accounting  Office  (1996)  to  be  inadequate. 
These  other  systems  include  the  following:  the  Joint  Readiness  Management  System  (Gillis,  1996),  Joint  Mission 
Essential  Task  Lists  (JMETL)  (Wagner,  1996),  Joint  Automated  Readiness  System  (JARS)  (Neal,  1996),  and  the 
Readiness  Baseline  Indicators  project  (Medlock,  1996).  Each  of  these  R&D  systems  offers  a  different  emphasis, 
including  the  ability  to  assess  the  readiness  to  perform  specific  missions  and  the  tasks  associated  with  those 
missions.  This  readiness  and  reporting  system  emphasizes  the  importance  of  relating  MOEs/MOPs  to  the 
mission. 

Operational  Readiness  Inspections  (ORIs).  The  famous  "We're  here  to  help."  ORI  visits  by  the  Inspector 
General  (IG)  result  in  a  report  often  used  to  justify  additional  funding  or  to  shift  resources  to  "fix"  problems.  API 
90-201,  Inspecior  General  Activities  (SAF/IGI,  1996)  describes  the  subjects  that  will  be  covered  during  ORIs. 
The  Air  Combat  Command  (ACC),  Inspector  General  (ACC/IGIX  1996),  has  published  supplements  to  this 
instruction  that  detail  specific  objective  grading  criteria,  while  other  MAJCOM  supplements  contain  more 
general  and  subjective  criteria  (AFSOC/IG,  1995;  AMC/IGPS,  1996). 

Command  Status  Briefings.  These  briefings  provide  MAJCOM  commanders  and  their  key  staff  highlights  of 
the  readiness  of  their  imits  by  weapon  system.  These  briefings  are  especially  important  since  they  contain  metrics 
which  measure  the  "health"  of  the  force  that  the  commanders  and  key  staff  members  have  requested 
(AMC/LGQP,  1996).  These  briefings  tell  us  their  priorities.  In  some  cases,  the  metrics  are  clearly  defined  in  a 
MAJCOM  pamphlet  (AMC/LGQA,  1995),  while  other  status  briefings  may  tend  to  be  more  variable,  be 
computed  by  staff  analysts  on  an  ad  hoc  basis,  and  lack  full  documentation. 

Military  Exercises.  Both  the  AF  and  the  Joint  Chiefs  of  Staff  conduct  exercises  on  a  continuing  basis  to  ensure 
the  force  is  ready  to  perform  a  variety  of  missions  and  contingencies.  Exercises  use  real  aircraft  and  simulated 
combat  situations  to  test  readiness  and  train  war-fighting  ability.  Some  last  a  few  hours  or  days,  while  others  are 
extended  over  a  longer  period.  Blue  (AF  News  Service  [AFNS],  1996a)  and  Red  Flag  (AF  Flight  Weapons 
Center  [AFFWCj/Public  Affairs  [PA],  1992)  are  examples  of  highly  visible  exercises;  however,  exercises  are 
numerous  and  most  are  classified.  Metrics  are  developed  to  aid  in  scoring  the  exercises;  however,  overall 
assessments  are  often  subjective. 

Wargames  and  Simulations.  Wargames  and  simulations  involve  replicating  warfare  without  actual  combat, 
often  involving  computer  simulations.  Distributed  simulation  technology  is  now  integrated  into  many  exercises, 
competitions,  and  training  programs.  While  the  basic  data  on  number  of  kills  per  mission  and  straightforward 
MOEs  are  readily  available,  detailed  metrics  are  difficult  to  find  in  the  literature  about  these  distributed 
simulations.  Wargames  and  model-type  simulations  include  BATMAN  &  RORZN  (Federico,  et  al.,  1991),  AF 
Commander,  Conflict:  Korea,  TAC  Brawler  (SurvivabilityA^ulnerability  Information  Analysis  Center 
[SURVIAC],  1994),  and  the  Logistics  Composite  Model  (LCOM)  (Clark,  1989;  Boyle,  1990).  The  latter  two  have 
also  been  used  during  systems  acquisition  to  optimize  MPT  requirements  with  other  weapon  system 
requirements  and  design  factors  on  aircraft  systems  like  the  F-22  and  Joint  Strike  Fighter. 

Military  Competitions.  Every  year  AF  teams  compete  for  top  honors  in  their  mission  areas.  Competitions 
appear  to  be  one  of  the  more  interesting  sources  of  MOEs/MOPs.  For  example,  ACC's  William  Tell 
(ACC/DOOO,  1996)  combines  competitions  for  pilot  and  weapons  director  accuracy,  munitions  loading  speed 
and  safety,  and  aircraft  maintenance  proficiency.  Other  examples  of  competitions  include  the  ACC-sponsored 
combined  force  competitions,  Gunsmoke  (AFFWC/PA,  1992)  and  combined  mid-  and  long-range  bombing 
competition.  Long  Shot  (AFNS,  1996b),  and  AMC's  transportation  tanker  and  airlift  Rodeo  (AMC/PA,  1996). 

Surrogate  Measures.  Surrogate  measures  are  often  developed  for  evaluation  of  exercises,  wargames, 
simulations,  competitions,  operational  tests,  ORIs.  Like  MOPs,  surrogate  measures  often  do  not  measure  the  foil 
MOE.  Despite  the  fact  that  surrogate  measures  cover  only  some  aspects  of  the  MOE  and  only  are  able  to 
approximate  the  MOE,  these  measures  are  often  considered  acceptable  indicators  of  readiness  conditions  by 
commanders  and  high-level  decision-makers.  Since  MAJCOM  decision-makers  are  often  willing  to  accept  these 
surrogate  measures  as  strong  indicators  of  war-time  outcomes,  and  since  the  surrogate  measures  may  be  more 
easily  measured  than  MOEs,  these  measures  may  be  more  readily  linked  with  MPT  research  and  effectiveness 
measures  than  MOEs.  Since  these  surrogate  measures  have  already  been  established  as  credible,  they  may 
convey  credibility  to  the  MPT  technology  being  tested. 

CONCLUSION  AND  RECOMMENDATIONS 
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Few  Formal  Comprehensive  MOE/MOP  Listings  Available.  Lack  of  comprehensive  aircraft-related 
MOEs/MOPs  listing  and  a  dearth  of  unclassified  scientific  articles  on  the  topic  forced  the  labor-intensive  use  of 
networking  to  identify  AF  aeronautical  MOEs/MOPs.  Considerable  information  was  available,  but  not  from 
widely  puWished  sources.  Two  exceptions  to  this  general  rule  were  (1)  MIL-STD-1776A,  which  contained  an 
extensive,  though  incomplete,  listing  of  operator/system-related  MOEs/MOPs,  and  (2)  API  10-602,  Attachment 
3,  which  contained  a  comprehensive  listing  of  maintenance  MOEs/MOPs.  Through  extensive  networking,  a 
number  of  other  good  sources  of  AF  MOEs/MOPs  were  identified.  Many  of  these  sources  are  not  formally 
published,  are  in  the  form  of  informal  guides  to  analysts,  or  are  published  as  MAJCOM  supplements  (not  widely 
available). 

MOE/MOP  Taxonomy  -  A  Work  In-Progress.  The  CSERIAC  Review  and  Analysis  (Best,  Gentner, 
Cunningham,  &  Schopper,  in  preparation)  documents  major  MOEs/MOPs  that  were  available  in  a  taxonomy, 
and  while  this  is  not  fully  comprehensive,  we  believe  that  it  contains  an  extensive  sample  of  MOEs/MOPs, 
together  with  typical  sources  of  MOEs/MOPs  and  a  process  for  deriving  them.  As  this  taxonomy  is  refined,  it 
should  assist  scientists  in  identifying  MOEs/MOPs  that  might  benefit  from  their  research. 

MAPs  and  Acquisition  Documents  Offer  Research  Targets.  Since  the  MAJCOMs  are  defining  deficiencies  in 
MAPs  and  criteria-for-success  in  acquisition  documents,  it  would  behoove  scientists  to  examine  these 
deficiencies  for  possible  research  projects  to  improve  human  performance  in  the  deficiency  areas.  If  MPT 
researchers  want  to  target  their  research  toward  a  new  weapon  system  or  major  modification,  they  should  consult 
the  ROM  of  major  systems  to  determine  how  system  performance  could  be  improved  through  improved  human 
performance. 

Remaining  Work.  Since  few  displays  of  MOE/MOP  hierarchies  are  found  in  the  literature,  it  appears  that  the 
most  difficult  part  of  this  study  will  be  to  develop  an  appropriate  hierarchy  for  the  taxonomy.  It  is  possible  that 
some  of  the  classified  and  limited  documents,  such  as  T&E  and  Cost  and  Effectiveness  Analyses  (COEAs), 
which  have  not  been  reviewed  to  this  point,  may  contain  hierarchical  listings  of  MOEs  which  could  be  helpful  in 
developing  ones  tailored  to  depict  common  aircraft-related  MOE/MOP  structures.  Based  on  the  findings  of  this 
phase,  a  structured  interview  survey  instrument  will  be  constructed,  and  high-level  decision-makers  will  be 
polled  about  their  MOE/MOP  priorities.  The  results  will  be  compared  with  the  written  documentation  on 
MOEs/MOPs,  and  adjustments  made  to  the  MOE/MOP  taxonomy.  MOFIMOP  priority  variances  will  be  noted. 

Recommendations  for  Remaining  Work 

Specific  recommendations  for  the  next  phase  include  the  following:  (1)  review  limited  distribution  documents 
(and  classified,  if  necessary)  to  develop  hierarchical  listings  of  AF  MOEs;  (2)  expand  the  MOE/MOP  taxonomy 
to  include  late-arriving  documents  requested  from  AF  sources;  (3)  review  deficiencies  documented  in  MAPs  to 
identify  current  issues  that  could  be  linked  with  human  performance  metrics;  (4)  develop  a  structured  interview 
and  conduct  a  survey  of  high-level  AF  decision-makers,  focusing  on  those  in  AF  offices  charged  with 
maintaining  war-fighting  capability  and  keeping  MAJCOM  commanders  informed  on  the  readiness  of  their 
forces;  and  (5)  focus  on  a  limited  number  of  mission  areas  where  MOEs/MOPs  are  documented  and  where 
persoimel  can  afford  time  to  assist  in  specifying  MOEs  and  MOPs  when  they  are  not  available  from  the 
literature. 
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